Papers by Seong Joon Oh
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) are being adopted more widely as personal agents thanks to their enhanced planning skills enabled by reasoning traces (RTs). |
| Approach: | They propose to increase the budget of models with increased reasoning steps to amplify such leakage by enlarging their internal thinking to the model's internal thinking. |
| Outcome: | The proposed model can reason more verbosely and leak more in their own thinking, while improving utility but enlarges the privacy attack surface. |
TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. |
| Approach: | They propose a method that uses adversarial suffixes to get an answer from a target LLM. |
| Outcome: | The proposed method detects the LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. |
Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning of frontier models can lead to privacy collapse, causing optimisation for helpfulness, exposure to user information, and debugging code printing internal variables. |
| Approach: | They propose to fine-tune frontier models to adapt to specific domains and align with organizational workflows and user preferences. |
| Outcome: | The proposed model fails to perform on safety and utility benchmarks while exhibiting severe privacy vulnerabilities. |
Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Membership inference attacks (MIAs) attempt to verify the membership of a data sample in the training set for a model. |
| Approach: | They propose to use membership inference attacks to verify the membership of a given data sample in a model training set. |
| Outcome: | The proposed methods are based on a new benchmark that measures the performance of membership inference attacks on large language models at a continuous scale. |
MASEval: Extending Multi-Agent Evaluation from Models to Systems (2026.acl-demo)
Copied to clipboard
Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl, Sangdoo Yun, Seong Joon Oh, Martin Gubri
| Challenge: | MASEval provides a framework-agnostic, system-level comparison across any agent framework and benchmark. |
| Approach: | They propose a Python library that treats the entire agentic system as the unit of analysis. |
| Outcome: | The proposed framework treats the entire agentic system as the unit of analysis. |